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Abstract 

This  study  is  a  first  step  in  formalizing  CRM  training  as  part  of 
Undergraduate  Space  and  Missile  Training  (USMT)  and  individual  Unit  Training  thereby 
capitalizing  on  techniques  already  developed  and  in  place  for  pilots.  To  facilitate  the 
integration  of  CRM  into  Space/  Missile  Crew  Training,  a  tool  must  be  created  to  measure 
CRM  concepts  and  test  their  relationships  with  individual  and  group  performance.  We 
evaluated  the  usefulness  of  two  modified  versions  of  existing  CRM  measures  for  use  by 
USMT,  Unit  and  Guardian  Challenge  instructors  or  evaluators.  Because  the  attitudes 
measure  is  easier  to  implement,  this  study  analyzes  the  use  of  an  established  flight  crew 
attitude  measure,  the  Cockpit  Management  Attitudes  Questionnaire  (CMAQ)  [Gregorich, 
1990],  to  determine  if  it  can  be  used  as  an  indicator  of  overall  Space  Operations  crew 
performance.  This  study  also  tests  a  modified  version  of  the  Air  Force  Mobility 
Command  AMC  Form  128,  March  1995.  We  conducted  the  study  in  two  phases.  The 
first  phase  dealt  only  with  Space  Operations  crews.  The  second  phase  dealt  with  the 
participants  of  the  Guardian  Challenge  Competition. 

Phase  1  correlations  between  an  individual’s  self  assessed  attitude  and  that 
individual’s  rating  by  his  superior  show  that  Assertiveness  with  Cronbach’s  alphas  of  .61 
and  .66  on  each  questionnaire  respectively,  and  a  correlation  of  .204  (significant  to  .04,  2- 
tailed)  is  the  only  reliably  measured  attitude  related  to  performance. 

Also,  phase  1  showed  that  graduates  of  Undergraduate  Space  and  Missile  Training 
consistently  had  better  CRM  related  attitudes  and  were  rated  higher  in  the  seven  Form 
128  areas  than  crewmen  who  did  not  attend  USMT. 


Phase  2  results  were  unexpected  but  extremely  consistent.  The  higher  a  crew’s 
score  in  Guardian  Challenge,  the  worse  the  crew  rated  itself  at  the  group  level  in  CRM 
attitudes. 

Generally,  the  data  collected  seem  to  indicate  that  beyond  using  Assertiveness, 
attitude  measures  are  a  poor  indicator  of  whether  or  not  an  individual  crewman  will 
successfully  utilize  CRM.  Therefore,  attitude  measurement  is  a  poor  performance 
indicator  in  the  Space/Missile  Operations  environment.  Stated  another  way,  the  CMAQ 
does  not  translate  well  into  Space/Missile  Operations  and  should  not  be  used  as  an 
assessment  tool  by  instructors.  The  Form  128  may  still  serve  as  a  guide  for  designing  a 
CRM  evaluation  sheet  for  Space/Missile  crews,  but  the  sheet  does  not  seem  well  suited 
as,  or  easily  converted  into  a  self-assessment  sheet  to  be  used  by  individuals. 

Overall,  the  Air  Force  needs  to  train  yoimg  enlisted  Space  Operations  crewmen  to 
be  more  assertive  with  senior  enlisted  and  officers. 


A  PILOT  STUDY  OF  SPACE/  MISSILE  CREWS  AND 
CREW  RESOURCE  MANAGEMENT 

1.  Introduction 

General  Issue 

As  technological  complexity  increases,  critical  tasks  are  more  likely  to  be 
accomplished  by  teams.  Teams,  of  specialists  with  different  responsibilities  and 
expertise,  routinely  work  together  to  perform  surgery,  operate  nuclear  reactors,  fly 
airplanes,  and  control  spacecraft  in  orbit.  In  each  of  these  professions,  errors  may  have 
life  or  death  consequences.  Each  profession  is  also  characterized  by  advanced 
technology,  a  rapid  operational  pace,  and  complexity  that  exceeds  individual  capacities. 
One  of  the  most  difficult  problems  facing  organizations  that  make  use  of  complex 
technologies  is  ensuring  that  specialists  communicate  and  coordinate  their  activities 
effectively  with  other  team  members.  Crew  Resource  Management  (CRM)  is  one 
approach  addressing  this  problem.  CRM  has  been  successfully  applied  to  several  high 
technology  professions  (i.e.  Airline/military  pilots,  merchant  mariners,  nuclear  power 
plant  operators),  but  to  date  no  attempt  has  been  made  to  adapt  CRM  for  use  by  Space/ 
Missile  Operators.  This  study  is  the  first  step  towards  introducing  CRM  techniques  to  the 
Space/  Missile  Operations  profession. 

Background 

Crew  Resource  Management  began  in  the  early  1980’s  as  Cockpit  Resource 
Management.  NASA  and  several  airlines  began  a  study  to  determine  why  “good”  pilots 
sometimes  crash  “good”  airplanes.  CRM  originally  focused  on  improving  the  way  airline 
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flight  crews  interact  with  their  environment,  their  machinery  and  most  importantly  with 
other  flight  crew  members.  These  studies  developed  into  several  training  programs  such 
as  NASA’s  Line  Oriented  Flight  Training  Program  (LOFT)  [Helmreich,  1990]  or  the 
former  Strategic  Air  Command’s  Mission  Oriented  Simulator  Training  (MOST) 

[Wiener,  1993].  These  programs  appear  successful  in  improving  the  overall  performance 
of  these  flight  crews  in  simulators  [Helmreich,  1990]  and  in  reducing  errors  during  actual 
missions  [Bates,  1997].  As  a  result,  researchers  have  become  interested  in  expanding  the 
use  of  CRM  to  other  crews  operating  complex  equipment  in  high  stakes  and  therefore 
high  stress  environments.  For  example,  recent  studies  of  CRM  have  examined  the 
interactions  of  surgical  teams  within  operating  rooms  [Helmreich,  1997]. 

Flying  mutli-pilot  aircraft  shares  many  characteristics  with  Space/  Missile 
Operations.  Both  professions  involve  the  control  of  high-technology,  highly-automated 
machines  that  if  operated  improperly  can  lead  to  massive  loss  of  life  and  property. 
Because  of  these  consequences,  each  profession  induces  a  tremendous  amount  of  stress 
upon  teams.  Each  of  these  systems  is  controlled  by  teams  who  interpret  and  react  to 
possibly  erroneous  and  conflicting  information  provided  by  remote  sensors.  One  might 
mistakenly  point  out  one  key  difference  between  pilots  and  space  operators:  unlike  space 
operators,  pilots  are  aboard  their  craft  and  therefore  can  use  their  senses  to  “feel”  what  the 
aircraft  is  doing.  As  anyone  who  has  ever  flown  an  aircraft  in  instrument  conditions  can 
tell  you,  the  aircraft’s  instruments  are  the  only  reliable  indicators  of  an  aircraft’s  attitude 
and  even  its  speed. 

Also,  in  both  professions,  different  team-members  are  provided  with  different 
pieces  of  a  puzzle  that  they  must  communicate  to  the  rest  of  their  crew.  For  instance,  it 
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is  common  practice  in  multi-pilot  aircraft  for  one  pilot  to  fly  the  aircraft  while  the  other 
pilot  operates  the  radios.  Often  the  flying  pilot  can  become  distracted  by  the  flying  task 
and  does  not  hear  instructions  from  ground  controllers.  The  non-flying  pilot  then  must 
communicate  the  controller’s  instructions  to  the  flying  pilot.  Similarly,  the  Space 
Operator  controlling  a  satellite  is  rarely  the  one  maintaining  the  communications  links 
between  his  control  center  and  his  remote  antenna.  If  this  link  fails  or  is  about  to  be 
moved  beyond  its  limits  of  travel,  the  Ground  System  Operator  is  responsible  for 
communicating  this  information  to  the  Space  Operator  to  prevent  him  from  missing 
critical  commands.  In  both  cases,  the  quality  of  this  communication,  and  assertiveness  of 
the  communicator  can  have  drastic  effects  on  the  pilot  or  operator’s  overall  Situational 
Awareness.  In  either  case,  a  lack  of  Situational  Awareness  could  lead  to  either  a  crashed 
airplane  or  a  ‘dead’  satellite. 

Currently,  there  are  two  philosophies  on  how  to  effectively  measure  a  crew’s  use 
of  CRM  concepts.  Both  philosophies  attempt  to  use  some  measure  of  crew  or  individual 
CRM  effectiveness  as  a  predictor  of  overall  job  performance.  The  first  philosophy  is 
based  upon  the  influence  of  attitudes  on  individual  behaviors  as  asserted  by  Ajzen  and 
Fishbein’s  model  [1975].  Attitudes  are  easily  measured  using  surveys  and  if  the  Fishbein 
model  works  well  in  this  context,  the  attitudes  indicated  by  the  survey  results  should  give 
some  indication  of  how  well  a  crewman  performs  in  terms  of  CRM.  This  is  the  basic 
philosophy  espoused  by  Gregorich,  et  al  [1990]  in  The  Structure  of  Cockpit  Management 
Attitudes.  Gregorich,  et  al.  believe  that  if  individual  attitudes  about  CRM  are  accurately 
measiued,  these  measurements  should  accurately  predict  an  individual  pilot’s  behaviors 
and  therefore  his  overall  performance  as  a  crewmember.  The  other  philosophy  is  held  by 
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the  research  teams  working  for  the  U.S.  Air  Force’s  Armstrong  Laboratory.  They  state 
that  “the  assumption  that  attitudes  directly  influence  behaviors  is  questionable. .  .Asa 
result,  requirements  for  measuring  CRM  skills  and  behaviors  have  emerged”  [Murray,  et 
al.  1995]  Essentially,  the  Armstrong  researchers  believe  in  the  development  of  a  Critical 
Incidents  Approach  to  measuring  CRM.  Summarizing,  operators  are  asked  to  tell  “war- 
stories”.  These  stories  are  then  distilled  into  basic  critical  behaviors  that  are  ranked  by 
the  operators  as  being  good  or  bad  for  CRM.  These  behaviors  are  then  used  by  evaluators 
to  rate  crews  on  how  well  they  perform  CRM.  Each  crew’s  rating  in  CRM  is  then 
compared  to  their  overall  rating  to  identify  which  CRM  behaviors  promote  high  overall 
crew  performance. 

Of  the  two  philosophies,  the  Attitudes  approach  by  far  requires  the  least 
expenditure  of  resources  to  implement  since  surveys  used  to  measure  attitudes  exist  and 
have  the  potential  to  be  applied  across  a  wide  range  of  specialties.  The  Critical  Incidents 
approach,  on  the  other  hand,  requires  a  great  deal  of  cooperation  and  time  from  the 
operators  to  be  measured,  just  to  create  the  basic  criterion.  The  other  disadvantage  is  that 
this  method  has  to  be  recreated  for  each  system  type.  In  other  words,  the  criteria  setup  to 
judge  the  CRM  effectiveness  of  a  bomber  crew  may  not  be  applicable  to  a  satellite 
operations  crew. 

Problem  Statement 

To  facilitate  the  integration  of  CRM  into  Space/  Missile  Crew  Training,  a  tool 
must  be  created  to  measure  CRM  concepts  and  test  their  relationships  with  individual  and 
group  performance.  Because  the  attitudes  measure  is  easier  to  implement,  this  study 
analyzes  the  use  of  an  established  attitude  measure,  the  Cockpit  Management  Attitudes 
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Questionnaire  (CMAQ)  [Gregorich,  1990],  to  determine  if  it  can  be  used  as  an  indicator 
of  overall  Space  Operations  crew  performance.  This  study  also  tests  a  modified  version 
of  the  Air  Force  Mobility  Command  AMC  Form  128,  March  1995. 

Research  Objectives 

The  overarching  objective  of  this  study  is  the  prevention  of  the  catastrophic  loss 
of  an  American  space  or  nuclear  asset  due  to  human  error.  This  goal  is  best  accomplished 
by  assisting  Air  Force  Space  Command  efforts  to  improve  crew  communication  and 
coordination.  This  study  is  a  first  step  in  formalizing  CRM  training  as  part  of 
Undergraduate  Space  and  Missile  Training  (USMT)  and  individual  Unit  Training  thereby 
capitalizing  on  techniques  already  developed  and  in  place  for  pilots.  This  study  evaluates 
the  usefulness  of  two  modified  versions  of  existing  CRM  measures  for  use  by  USMT, 
Unit  and  Guardian  Challenge  instructors  or  evaluators. 
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II.  Literature  Review 


Measuring  Team  Performance 

The  first  step  in  studying  crews  is  to  determine  the  type  of  team/  crew  to  be 
studied.  According  to  Brannick,  et  al.  [1997],  their  are  four  basic  types  of  teams;  Pooled/ 
Additive,  Sequential,  Reciprocal,  and  Intensive.  Brannick  describes  a  Pooled /Additive 
team  as  the  most  basic  form  of  team.  Pooled/  Additive  teams  accomplish  work 
independently,  but  their  group  effect  is  measured.  For  instance,  the  members  of  a 
janitorial  team  work  independently  to  clean  a  building.  Each  member’s  efforts  are 
performed  independently  of  the  rest  of  the  team’s.  Sequential  teams  are  best  described  by 
an  assembly  line.  Each  member’s  actions  begin  when  the  previous  person  is  finished. 
Again,  each  member’s  individual  tasks  are  independent  of  the  previous  members. 
Reciprocal  teams  are  a  permutation  of  sequential  teams.  Reciprocal  teams  are  best 
described  by  an  assembly  line  where  the  product  is  moved  predictably  back  and  forth 
between  several  members  imtil  the  task  is  completed.  The  Intensive  team,  on  the  other 
hand  has  no  set  sequence  of  predictable  events.  Each  member  participates  as  they  are 
needed  and  are  what  Schmitt,  et  al.  [1991]  referred  to  as  fate  interdependent.  Generally, 
every  task  requires  the  participation  of  several  team  members  to  be  accomplished. 
Intensive  teams  best  describe  the  activities  of  Operating  Room  staffs  [Brannick,  et  al., 
1997]  or  flight  crews.  Like  an  Operating  Room  team.  Space  Operations  crews  are  an 
Intensive  team  arrangement.  Both  types  of  teams  require  a  “mutual  situational  awareness, 
where  team  members  are  able  to  predict,  adapt,  and  coordinate  with  one  another 
successfully,  even  imder  stressful  or  novel  conditions”  [Brannick,  et  al.,  1997]. 
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The  second  step  is  the  identification  of  critical  lexers  [Brannick,  et  al.,  1997]. 
“Critical  levers  [are]  the  most  important  factors  or  work  processes  that  underlie  a 
particular  team’s  performance”  [Brannick,  et  al.,  1997].  The  critical  levers  measured  by 
the  current  study  are  collectively  known  as  Crew  Resource  Management. 

The  third  step  to  studying  crews  is  to  link  the  critical  levers  to  a  measurement 
strategy  [Braimick,  et  al.,  1997].  Researchers  need  to  choose  their  “source  of 
information”  and  their  “method  of  measurement”  [Brannick  et  al.  1997].  “Sources  may 
be  either  incumbents ...  or  observers  of  some  sort.  The  most  common  source  . .  .has  been 
[the]  team  members  themselves”  [Brannick,  et  al.,  1997].  Team  members  are  best 
positioned  to  understand  the  nuances  of  subtle  communication  cues  and  how  these 
interactions  affect  a  given  task. 

The  last  step  is  the  measurement  and  aggregation  of  team  results.  “Data  on  team 
. . .  communication,  and  coordination. . .  are  often  collected  through  members’  ratings. 
These  scores  ...  are  typically  then  aggregated  . .  .to  form  a  single  [team]  score”  [Brannick 
etal.,  1997]. 

Introduction  to  CRM 

“CRM  includes  optimizing  not  only  the  person-machine  interface  and  the 
acquisition  of  timely,  appropriate  information,  but  also  interpersonal  activities  including 
leadership,  effective  team  formation  and  maintenance,  problem-solving,  decision  making, 
and  maintaining  situational  awareness”  [Wiener,  1993:4].  Succinctly,  CRM  is  the  use  of 
all  available  resources  to  get  the  job  done  safely  and  effectively. 
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CRM  and  Critical  Levers 


CRM  consists  of  several  of  what  Brannick  calls  critical  levers.  Various  authors 
have  distilled  CRM  into  different,  but  generally  similar  skill  areas  or  critical  levers. 
NASA  has  performed  most  of  its  work  in  seven  areas:  Communication,  Situational 
Awareness,  Problem  solving/  Decision  Making/  Judgement,  Leadership/  Followership, 
Stress  Management ,  Self  and  Peer  Critiquing,  and  Interpersonal  Skills  [Orlady,  et  al., 
1987].  Gregorich,  et  al.  [1990]  chose  to  focus  on  measuring  three  areas  they  believed 
were  key  to  CRM:  Communication  and  Coordination,  Command  Responsibility,  and  the 
Recognition  of  Stressor  Effects.  More  recent  work  with  MC- 1  SOP  aircrews  by 
Armstrong  Laboratory  divided  CRM  into  four  areas:  Fxmction  Allocation  (division  of 
labor).  Tactics  Employment,  Situation  Awareness,  Command/  Control/  Communications, 
and  Time  Management  [Silverman,  et  al.,  1998].  Each  of  these  research  groups  focused 
on  what  can  be  called  the  core  constructs  of  CRM:  Communication  and  Command  and 
what  most  researchers  identify  as  crew  and  individual  Situational  Awareness.  The  rest  of 
the  areas  measured  by  researchers  are  guided  by  the  seven  factors  identified  in  the 
original  NASA  research. 

CRM  and  Individual  Attitudes 

Two  key  aspects  of  CRM  are  an  operator’s  attitude  about  CRM  constructs  and 
how  those  attitudes  are  related  to  individual  and  team  performance.  Traditional  CRM 
research  on  pilots  has  taken  a  two-pronged  approach  patterned  after  Helmreich,  et  al’s 
[1990]  work.  The  first  step  is  to  measure  the  attitudes  of  the  pilots  using  a  survey  such  as 
the  CMAQ.  The  second  step  is  to  measure  their  performance/behaviors  during  actual 
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flight,  or  during  intense  sessions  in  a  realistic  flight  simulator.  The  simulator  is  essential 
since  this  is  the  crucible  where  pilots:  1.  demonstrate  the  application  of  CRM,  2.  their 
performance  can  be  efficiently  graded  by  instructors,  and  3.  where  a  pilot’s  failure  does 
not  precipitate  loss  of  life.  The  simulator  is  also  a  more  accurate  measuring  device  since 
the  scenario  can  be  controlled  by  the  rater/observer  and  pilot  performance  on  the  same 
tasks  can  be  accurately  compared.  Lastly,  researchers  correlate  the  attitudes  with  the 
behaviors  observed  in  the  simulators.  Significant  correlations  give  evidence  that  the 
positive  CRM  attitudes  are  associated  with  good  performance  [Helmreich,  1990]. 

History  of  the  Cockpit  Management  Attitudes  Questionnaire  (CMAQ) 

The  questionnaire  used  in  this  study  is  adapted  from  Gregorich,  Helmreich  and 
Wilhelm’s  [1990]  CMAQ.  The  CMAQ  consists  of  25  items.  Using  a  factor  analysis  with 
an  oblique  rotation,  Gregorich  et  al.  [1990]  identified  three  CRM  factors:  Communication 
and  Coordination,  Command  Responsibility  and  the  Recognition  of  Stressor  Effects.  A 
fourth  factor  known  as  Avoidance  of  Interpersonal  Conflict  was  shown  to  be  inconsistent 
among  the  pilots  used  in  the  survey.  Also,  of  the  25  questions,  19  proved  to  be  consistent 
identifiers  of  the  three  key  factors  listed  above. 

Air  Mobility  Command  Crew  Resource  Management  Assessment  Sheet 

The  Air  Mobility  Command  Crew  Resource  Management  Assessment  Sheet  is 
also  called  the  AMC  Form  128  (Form  128  for  short)  and  has  been  in  use  in  its  present 
form  since  March  1995.  The  Form  128  is  used  by  AMC  instructors  as  a  rating  sheet  to 
grade  flight  crews  on  their  use  of  CRM. 

The  Form  128  was  used  by  this  study  in  addition  to  the  CMAQ  since  it  covers 
many  of  the  CRM  topics  omitted  by  the  circa  1990  CMAQ.  Consisting  of  29  basic  items. 
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the  Form  128  provides  evaluators  guidance  in  rating  aircrews  on  Group  Dynamics, 
Effective  Communications,  Assertiveness,  Decision  Making,  Stress  Management,  Mishap 
Prevention  and  Overall  Observations  at  the  crew  or  individual  level.  The  Form  128  is 
also  similar  to  the  Line/  LOS  Checklist  for  Check  Pilots  developed  by  the  University  of 
Texas  Crew  Research  Project  [Wilhelm,  1996].  The  Form  128  was  chosen  for  this  study 
because  it  is  general  enough  to  be  easily  adapted  to  Space/  Missile  Operations  in  a 
military  environment. 

Space  and  Missile  Crews 

The  size,  training  and  make  up  of  space  operations  crews  vary  as  greatly  as  their 
spacecraft’s/  weapon  systems’  missions.  There  are  three  basic  types  of  Space  and  Missile 
crews:  Space  Operations,  Space  Lift,  and  Missile  Operations 

Space  Operations  Crews 

For  the  most  part,  space  operations  crews  remotely  control  satellites  from  ground 
control  centers.  Crew  sizes  range  from  4  up  to  as  many  as  13  crewman  depending  on  the 
system  they  operate.  Crew  shifts  vary  from  8  hours  to  12+  hours  on  a  rotating  shift  basis. 
Some  crews  work  six  days  on  with  three  to  four  days  off  before  changing  shifts  while 
others  change  shifts  every  three  days.  These  cycles  vary  greatly  between  different 
squadrons  and  change  as  often  as  the  squadrons  change  commanders. 

The  crews  are  generally  made  up  of  a  Crew  Commander,  Crew  Chief,  Satellite 
Systems  Operator,  Payload  Specialist,  Bus  Specialist  and  a  Ground  System  Operator. 

The  Commander  and  the  two  specialists  are  usually  manned  by  junior  officers.  The  Crew 
Chief  is  senior  enlisted  while  the  two  operators  are  junior  enlisted  men. 
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Space  Operations  crews  control  satellites  ranging  in  value  from  $35  million  to  $1 
billion  and  upload  and  download  data  and  position  and  reposition  satellites  used  for 
navigation,  communication,  missile  detection  and  early  warning,  and  weather  prediction 
for  both  civilian  and  military  users. 

Space  Operations  Training.  Space  Operations  officers  attend  Undergraduate 
Space  and  Missile  Training  (USMT)  at  Vandenberg  AFB,  CA  with  the  exception  of 
degreed  Aero/  Astronautical  Engineers  or  Electrical  Engineers  assigned  to  squadron 
engineering  positions.  The  Engineers  report  directly  to  system  specific  schools.  The 
enlisted  men  attend  a  Space  Operations  Technical  School  also  at  Vandenberg.  After 
graduation,  each  group  attends  a  system  specific  school  and  squadron  training  to  teach 
them  their  specific  duties. 

Daily  Space  Operations.  Generally  the  satellites  fly  on  automatic  pilot.  The 
crews  are  responsible  for  monitoring  the  automated  systems  and  for  reprogramming  the 
payload  computers  (uploads).  Each  unit  uses  a  worldwide  network  of  ground  antennas 
(10m  to  30m  parabolic  antennas)  to  monitor  and  command  its  satellites.  These  antennas 
are  a  shared  resource  making  resource  scheduling  between  units  a  difficult  task. 
Occasionally,  the  satellites  require  orbital  corrections  to  keep  them  in  place  over  the 
planet.  These  corrections  are  planning  intensive  and  if  done  improperly  can  result  in  the 
loss  of  a  satellite.  Also,  solar  flares  and  high  energy  storms  can  cause  automatic  systems 
on  the  satellites  to  fail  causing  the  crew  to  scramble  for  ground  resources  to  get  in  contact 
with  the  satellite,  figure  out  what  failed,  and  bypass  the  failed  systems  before  the  satellite 
nins  out  of  electricity:  effectively  “dying”  in  space. 
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Space  Lift  Crews 

Space  Lift  Crews  launch  rockets  into  space.  These  rockets  cost  between  $30 
million  to  $300  million  excluding  the  cost  of  their  payload.  The  crews  generally  consist 
of  Air  Force  Launch  Control,  Assistant  Launch  Control,  Bus  Operations  Control, 
Spacecraft  Operations  Control,  Upper-Stage  Operations  Control  and  a  Range  Control 
Officer.  Each  of  these  positions  is  manned  by  an  officer.  Some  launch  systems  have 
larger  crews  than  others.  Most  systems  use  civilian  contractors  to  back-up  each  of  these 
positions. 

Space  Lift  Training.  Space  Lift  Officers  undergo  the  same  basic  stages  of 
training  as  Space  Operations  crews.  However,  due  to  the  “one-shot”  nature  of  their 
business,  they  spend  the  majority  of  their  time  rehearsing  for  launches.  These  rehearsals 
are  performed  on  the  actual  laimch  equipment  using  a  combination  of  tape  playbacks  and 
‘paper  inputs’.  A  tape  playback  is  basically  a  recording  of  a  previous  launch’s  telemetry 
that  is  fed  back  into  the  crews  control  stations.  Situations  (anomalies)  are  introduced  into 
the  scenario  by  ‘paper  inputs’  handed  to  controllers  by  instructors/  evaluators.  In  this 
way,  launch  crews  are  taught  how  to  reason  their  way  through  an  emergency  and  practice 
coordination  between  the  positions. 

Daily  Space  Lift  Operations.  The  majority  of  a  Space  Lift  officer’s  time  is  spent 
planning  and  training  for  the  next  launch.  Some  units  launch  monthly  while  others 
launch  once  every  six  months.  This  study  focuses  on  the  activities  of  Space  Lift  crews 
during  a  countdown/  launch  operation. 

Space  lift  crews  perform  the  last  minute  system  checks  necessary  to  launch  a 
rocket  into  space.  They  utilize  a  large  script  sometimes  called  the  “Coimtdown  Manual- 
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Lavinch  Activities”.  This  script  provides  the  exact  sequence  to  configure  the  range, 
Launch  Facility,  and  Space  Lift  vehicle  for  launch.  Essentially,  anyone  who  has  watched 
NASA  television,  or  even  seen  the  film  Apollo  13,  has  a  basic  understanding  of  how  a 
Space  Lift  crew  works. 

As  a  rule,  launch  vehicles  can  not  be  turned  off  once  they  have  ignited.  This  fact 
when  added  to  the  counteracting  pressure  to  launch  on  schedule,  places  tremendous  stress 
and  responsibility  on  Space  Lift  crews  to  perform  perfectly,  eveiy  time. 

Missile  Operations  Crews 

Missile  Operations  Crews  consist  of  a  Crew  Commander  and  a  Deputy 
Commander.  They  are  both  officers  and  generally  work  24  hour  shifts  in  their 
underground  Launch  Control  Centers  (LCC).  However,  their  shifts  actually  begin  long 
before  they  arrive  at  the  LCC. 

Daily  Missile  Operations.  First  they  report  to  their  base  to  be  briefed  by  then- 
unit.  Then  they  travel  as  much  as  100  miles  to  their  LCC.  Then  their  24  hour  shift 
officially  begins.  Their  daily  responsibilities  include  managing  message  traffic  from  their 
command  post  and  overseeing  security  and  maintenance  of  their  flight  of  intercontinental 
ballistic  missiles  (ICBMs).  Should  the  final  order  come,  missile  crews  are  charged  with 
programming  targets  into  the  missiles  and  laimching  multiple  nuclear  warheads. 

Missile  Operations  Training.  Missile  crews  go  through  the  same  basic  training 
sequence  as  space  operations  crews.  Missile  crews  first  attend  USMT.  Then  they  attend 
a  Combat  Crew  Training  squadron  at  Vandenberg  for  system-specific  training.  Lastly, 
they  attend  unit  specific  training. 
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Guardian  Challenge 

Guardian  Challenge  is  a  yearly  competition  for  Air  Force  Space/  Missile  crews. 
Every  operations  squadron  in  Air  Force  Space  Command  puts  forth  one  crew  to  compete 
against  other  units  in  their  Wing.  The  finalists  from  each  Wing  compete  in  late  April  or 
early  May  every  year.  Units  compete  in  one  of  three  categories:  Space  Operations,  Space 
Lift  or  Missile  Operations. 

Space  Operations  and  Space  Lift  crews  compete  at  their  home  base  and  are  rated 
by  their  unit  standardization  and  evaluation  sections  (DOV).  The  use  of  unit  raters  is 
unavoidable  due  to  the  specific  training  required  to  operate  each  system.  However,  the 
raters  are  considered  ‘trusted  agents’  and  risk  serious  censure  if  they  were  to  provide  an 
unrealistic  scenario  to  their  crews.  Also,  the  crews  usually  compete  on  actual  equipment 
since  Space  Operators  have  no  effective  simulation  capability. 

Unlike  Space  and  Space  Lift  Crews,  Missile  Crews  actually  travel  to  Vandenberg 
AFB  and  compete  in  very  realistic  LCC  simulators.  They  are  rated  by  Guardian 
Challenge  evaluators.  Therefore,  of  the  three  types  of  crews  competing  at  Guardian 
Challenge,  Missile  crews  have  the  most  standardized  and  realistic  scenarios. 

However,  all  scenarios  created  by  the  various  evaluation  teams  undergo  central 
review  to  ensure  they  are  somewhat  standardized  for  difficulty. 

Crew  and  Individual  Performance  Measurement 

The  most  difficult  aspect  of  any  team  oriented  study  is  deciding  how  to  measure 
the  individual  contributions  of  crewmembers  that  improve  overall  team  performance. 

CRM  researchers  appear  to  be  split  on  whether  CRM  should  be  measured  at  the 
individual  level  as  proposed  by  Gregorich,  et  al.  [1990]  or  at  the  team  level  as  proposed 
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by  Murray,  et  al.  [  1 995].  The  key  to  this  debate  is  driven  by  how  a  researcher  chooses  to 
measure  performance.  In  order  to  capitalize  upon  the  strengths  of  either  approach,  we 
conducted  our  study  in  two  Phases.  The  first  Phase  addresses  the  individual’s 
contribution  to  CRM.  To  do  this  we  paired  the  individual  responses  of  crewman  to  their 
individual  performance  in  CRM  as  rated  by  their  direct  superiors.  Since  their  superiors 
interact  daily  with  these  individuals  in  the  crew  environment,  we  believe  the  raters 
provided  us  with  a  more  accurate  overall  assessment  than  could  be  devised  by  a  team  of 
researchers. 

To  address  that  part  of  the  CRM  community  that  believes  in  measuring  teams  as  a 
whole,  we  capitalized  on  the  Guardian  Challenge  competition.  In  a  “situation  of . . .  task 
interdependency”  such  as  a  team  competition,  research  into  the  “contributions  of 
individuals  would  be  inappropriate”  [Schmitt,  et  al.,  1991].  By  utilizing  the  same 
measuring  instrument  as  Phase  1,  and  aggregating  the  responses  into  a  team  response 
[Brannick,  et  al.,  1997],  we  believe  we  were  able  to  estimate  the  overall  CRM  attitude  of 
a  given  crew.  This  aggregate  could  then  be  compared  against  the  crews  overall 
competition  rating  to  determine  which  attitudes  seemed  to  actually  improve  crew 
performance. 
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III.  Phase  1:  Space  Operations  Crews  Methodology  and  Results 

We  conducted  the  study  in  two  phases.  The  first  phase  dealt  only  with  Space 
Operations  crews.  The  second  phase  dealt  with  the  participants  of  the  Guardian 
Challenge  Competition. 

Phase  1  Experimental  Goals 

The  basic  goal  of  Phase  1  was  to  identify  those  attitudes  held  by  Space  Operators 
that  correspond  to  their  overall  effectiveness/  performance  as  judged  by  their  superiors/ 
raters. 

Phase  1  Experimental  Design 

The  experiment  consisted  of  surveying  space  operators  to  measure  individual 
attitudes  and  perceptions  about  CRM  related  constructs  and  comparing  these  with 
individual  ratings  provided  by  each  operators  superior  to  assess  individual  performance. 
The  results  of  these  two  surveys  were  then  compared  to  identify  if  any  of  the  CRM 
attitudes  or  perceptions  measured  by  the  flying  community  are  useful  for  predicting 
performance  in  the  space  community. 

Phase  1  Experimental  Instruments 

Space  Operations  Crew  Resource  Management  Questionnaire 
The  first  step  in  the  experiment  was  to  measure  each  crewman’s  attitude  about 
different  aspects  of  CRM  in  the  same  way  as  Gregorich,  et  al.  [1990]  did.  To  do  this  we 
developed  the  Space  Operations  Crew  Resource  Management  Questioimaire  (SOCRMQ) 
which  can  be  foimd  in  Appendix  A.  It  consisted  of  75  items  presented  in  a  5  point  Likert 
scale  format  and  divided  into  three  parts:  background,  modified  CMAQ,  modified  Form 
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128.  The  1 1  background  items  gathered  general  information  about  an  individuals  rank, 
time  in  service,  time  in  position  and  education. 

The  next  25  items  were  adapted  from  the  1990  CMAQ  and  were  slightly  modified 
to  be  applicable  to  the  Space  Operations  environment.  For  example,  terms  such  as 
Captain  or  flight  deck  were  replaced  by  Flight  Commander  (now  Crew  Commander)  and 
Operations  Floor  respectively.  Great  care  was  taken  to  preserve  the  concept  of  each 
question  while  providing  a  simple  translation  between  flight  terminology  and  space 
terminology.  Two  items  were  added  to  the  end  of  the  CMAQ  addressing  checklist 
discipline  and  the  pre-satellite-contact  (pre-pass)  briefing. 

The  AMC  Form  128  underwent  a  more  dramatic  change.  The  Form  128  is 
designed  as  a  rating  sheet.  This  format  was  restructured  from  a  7  subject  area,  28  item 
rating  sheet,  into  a  7  subject  area,  34  item  questionnaire.  The  increase  in  the  number  of 
items  is  due  to  the  rating  sheet’s  use  of  multiple  questions  in  many  of  its  rating  items. 
These  questions  were  broken  out  into  separate  items.  Again,  great  care  was  taken  to 
minimize  changes  to  the  items.  For  the  most  part,  the  items  were  transitioned  verbatim 
making  translation  changes  where  needed  (i.e.  Captain  to  Commander).  The  seven 
subject  areas  were  Group  Dynamics,  Effective  Communications,  Assertiveness,  Decision 
Making,  Stress  Management,  Mishap  Prevention  and  Overall  Observations.  Each  of 
these  areas  easily  transferred  into  the  Space  Operations  environment  except  Mishap 
Prevention. 

On  the  Form  128,  Mishap  Prevention  is  a  combination  of  Situational  Awareness 
and  the  management  of  aircraft  automated  systems.  Generally,  pilots  are  taught  to 
disengage  automated  systems  if  the  system  produces  inputs  contradictory  to  the  pilot’s 


17 


wishes.  Satellites  can  not  be  hand  flown  so  it  is  impossible  to  disconnect  these  types  of 
systems.  These  items  were  modified  slightly  from  their  original  wording  in  an  attempt  to 
make  them  more  compatible  with  satellite  operation. 

Lastly,  there  are  five  questions  pertaining  to  spending  off-duty  time  with  fellow 
crewmembers,  original  certification  level,  most  recent  re-certification  level  and  whether 
or  not  the  operator  had  ever  caused  a  real-world  critical  error. 

The  Form  128  portion  of  the  SOCRMQ  (ratee)  measures  individual  perceptions 
about  the  performance  of  that  individual’s  crew  with  respect  to  CRM. 

Space  Operations  Rater’s  Crew  Resource  Management  Questionnaire 
The  second  step  in  Phase  1  was  to  document  how  each  supervisor/rater  rated  the 
effectiveness  of  each  crewman  in  CRM  terms.  To  do  this,  we  developed  the  Space 
Operations  Rater’s  Crew  Resource  Management  Questionnaire  (SORCRMQ)  which  can 
be  found  in  Appendix  B.  The  Rater’s  Questionnaire  is  drawn  directly  from  the  AMC 
Form  128.  Several  questions  pertaining  to  the  use  of  automated  systems  were  omitted  for 
the  reasons  stated  above.  Again,  the  form  was  modified  to  use  a  Likert  5  point  scale  and 
was  translated  to  space  operations  terminology. 

Phase  1  Experiment 

The  SOCRMQ  (ratee)  were  distributed  to  every  Space  Operator  currently  assigned 
to  a  crew  in  the  chosen  Group  at  a  chosen  Air  Force  Base.  Due  to  Air  Force  Regulations, 
participation  in  this  study  was  strictly  voluntary  and  if  desired,  anonymous. 

Simultaneously,  the  SORCRMQ  (rater)  was  distributed  to  every  Crew 
Commander  and  the  squadron  Director  of  Operations  (DO).  The  Crew  Commanders 
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rated  every  person  on  their  crew  and  each  squadron’s  DO  rated  the  Crew  Commanders. 
The  surveys  were  individually  sealed  by  each  participant  and  returned. 

Upon  receipt  of  the  data,  we  matched  ratings  to  individuals  using  names  (if 
provided)  or  Squadron,  Flight,  Position  and  Rank.  The  latter  method  proving  very 
reliable  when  individuals  chose  to  withhold  their  names.  We  then  loaded  the  data  into 
the  SPSS  version  8.0  statistical  program  for  analysis. 

Initially,  we  ran  a  reliability  analysis  on  the  CMAQ  portion  of  the  SOCRMQ 
survey  using  the  item  groupings  identified  in  Gregorich,  et  al’s  The  Structure  of  Cockpit 
Management  Attitudes  [1990].  The  reliabilities  obtained  were  deemed  too  low  with 
Cronbach  Alphas  of  .51  for  Communication  and  Coordination,  .26  for  Command 
Responsibility,  and  .28  for  Recognition  of  Stressor  Effects.  Clearly,  a  different  construct 
was  at  work.  We  ran  a  Factor  Analysis  using  Principal  Axis  Factoring  and  rotated  using 
Oblimin  with  Kaiser  Normalization  similar  to  the  method  used  by  Gregorich,  et  al.  [1990] 
when  they  originally  developed  the  CMAQ.  This  method  created  three  new  factors; 
Coordination  &  Communication  (a  =  .58),  Monitoring  (a  =  .47)  and  Briefings  (a  =  .56). 
These  reliabilities  are  in  line  with  Gregorich’ s  work. 

We  then  broke  down  the  AMC  Form  128  portion  of  the  SOCRMQ  into  the  seven 
Form  128  areas:  Group  Dynamics,  Communication,  Assertiveness,  Decision  Making, 
Stress  Management,  Mishap  Prevention,  and  an  Overall  rating.  The  reliabilities  for  these 
areas  were  decent  with  the  exception  of  Communication.  They  were  a  =  .72,  .49,  .66, 

.63,  .78,  .76  and  .71  respectively.  Communication  was  dropped  from  any  further  analysis 
due  to  its  low  reliability  of  a  =  .49. 
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The  SORCRMQ  was  analyzed  in  a  similar  fashion.  Again  we  performed  a  Factor 
Analysis  and  obtained  three  new  Factors.  The  first  factor  dealt  with  how  well  a  crewman 
actively  works  to  avoid  overloading  himself  This  factor  became  known  as  Overload  and 
had  a  reliability  of  a  =  .88.  The  second  factor  dealt  with  how  well  a  crewman 
communicated  his  workload  to  other  crewmen.  This  factor  became  known  as  Workload 
and  had  a  reliability  of  a  =  .81 .  The  last  factor  was  Communication  and  it  had  a 
reliability  of  a  =  .73.  Like  the  SOCRMQ,  we  divided  the  SORCRMQ  into  the  seven 
areas  defined  by  the  From  128.  These  areas  and  corresponding  reliabilities  were  Group 
Dynamics  (a  =  .61),  Communication  (a  =  .62),  Assertiveness  (a  =  .61),  Decision  Making 
(a  =  .52),  Stress  Management  (a  =  .83),  Mishap  Prevention  (a  =  .58)  and  an  Overall 
rating  (a  =  .74). 

Due  to  the  great  variability  in  how  Crew  Commanders  rated  individuals,  every 
rating  area  was  normalized  to  every  rater  using  a  standard  Z  distribution  formula 
arbitrarily  centered  on  50  and  multiplying  the  z-score  by  10  for  readability: 


Where  x  is  the  raw  sum  of  every  item  per  area,  p  is  the  average  of  this  score  for  a 
given  rater  and  a  is  the  standard  deviation  of  the  raw  score  for  any  given  rater.  Therefore 
X  is  the  normalized  score.  The  standardized  ratings  are  all  prefaced  with  an  ‘s’  in  the 
results  section  to  delineate  them  Ifrom  raw  scores.  The  basic  assumption  of  this 
application  of  standardization  is  that  every  crew  is  equally  blessed  \vith  good  performers 
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and  bad  performers.  This  is  a  good  assumption  since  squadrons  can  not  pick  and  choose 
who  is  assigned  to  them  via  the  Air  Force  Personnel  Center  (AFPC). 

Phase  1  Results 

In  all,  20  crews  were  surveyed  with  132  out  of  an  approximate  population  of  160 
crewmen  or  82.5%  responding.  In  addition  25  raters  were  surveyed  (20  Crew 
Commanders  +  5  DO’s)  with  20  or  80%  responding.  Between  these  two  groups,  we 
received  attitude  measures  and  performance  ratings  for  102  or  64%  of  the  crewmen. 
Since  some  crewmen  did  not  answer  every  question,  some  items  had  a  lower  response 
rate  with  the  lowest  rate  being  59%.  Thus,  since  at  minimum  59%  of  the  Space 
Operations  population  responded,  these  results  provide  a  strong  indication  of  which 
individual  CRM  attitudes  affect  performance.  Also,  due  to  the  high  response  rate,  we 
chose  not  to  perform  a  Bonferroni  analysis. 

A  Pearson  Correlation  of  the  seven  Form  128  divisions  between  the  SOCRMQ 
and  the  standardized  SORCRMQ  sans  Communication  yielded  two  significant 
correlations.  First,  Assertiveness  had  a  .204  correlation  (significant  to  .04, 2-tailed) 
between  the  two  surveys.  Second,  the  Mishap  Prevention  attitude  (SOCRMQ)  and  Stress 
Management  performance  (SORCRMQ)  had  a  correlation  of  .173  (significant  to  .08, 2- 
tailed). 
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Table  1.  Phase  1  Results 


SGRPDYN  SCOMEFF  SASSERT  SDECMAK  SSTRESS  SMISHAP  SOVERALL 


COORD  Pearson 

.071 

.044 

.028 

-.003 

.001 

.049 

.006 

Correlation 

Sig.  (2-tailed) 

.473 

.657 

.782 

.973 

.993 

.622 

.954 

N 

103 

103 

103 

103 

103 

103 

98 

MONITOR  Pearson 

-.048 

.004 

.000 

.040 

.002 

.096 

-.029 

Correlation 

Sig.  (2-tailed) 

.630 

.969 

.998 

.687 

.985 

.334 

.778 

N 

103 

103 

103 

103 

103 

103 

98 

BRIEF  Pearson 

-.135 

-.057 

.056 

-.134 

-.070 

-.074 

-.084 

Correlation 

Sig.  (2-tailed) 

.175 

.566 

.573 

.180 

.482 

.459 

.415 

N 

102 

102 

102 

102 

102 

102 

97 

GRPDYN  Pearson 

.135 

.103 

.119 

.062 

.119 

.027 

-.002 

Correlation 

Sig.  (2-talled) 

.176 

.301 

.233 

.536 

.234 

.788 

.982 

N 

102 

102 

102 

102 

102 

102 

97 

ASSERT  Pearson 

.093 

.100 

1204 

.007 

.023 

.145 

.064 

Correlation 

Sig.  (2-tailed) 

.355 

.322 

.041 

.944 

.823 

.149 

.535 

N 

101 

101 

101 

101 

101 

101 

96 

DECMAK  Pearson 

.103 

.046 

.126 

.015 

.056 

.025 

.001 

Correlation 

Sig.  (2-tailed) 

.301 

.649 

.206 

.882 

.578 

.806 

.993 

N 

102 

102 

102 

102 

102 

102 

97 

STRESMGT  Pearson 

.111 

.066 

.129 

.017 

.062 

.042 

.111 

Correlation 

Sig.  (2-tailed) 

.266 

.512 

.195 

.862 

.534 

.675 

.280 

N 

102 

102 

102 

102 

102 

102 

97 

MISHAP  Pearson 

.010 

-.028 

-.032 

-.031 

mz 

-.005 

.038 

Correlation 

Sig.  (2-tailed) 

.924 

.786 

.756 

.761 

.086 

.964 

.713 

N 

99 

99 

99 

99 

99 

99 

95 

OVERALL  Pearson 

.058 

-.034 

.015 

-.078 

-.001 

.005 

-.048 

Correlation 

Sig.  (2-tailed) 

.562 

.737 

.882 

.441 

.989 

.962 

.641 

N 

101 

101 

101 

101 

101 

101 

96 

OFFDUTY  Pearson 

.114 

.058 

.074 

.001 

.078 

-.029 

-.035 

Correlation 

Sig.  (2-talled) 

.260 

.566 

.462 

.993 

.442 

.773 

.736 

N 

100 

100 

100 

100 

100 

100 

95 

Correlations  between  the  three  performance  factors  derived  from  a  factor  analysis 
of  the  SORCRMQ  and  the  seven  Form  128  areas  of  the  SOCRMQ  were  not  significant. 

Correlations  between  the  three  attitude  factors  derived  from  factor  analysis  of  the 
SOCRMQ  and  the  seven  Form  128  areas  of  the  SORCMQ  were  also  not  significant. 

We  also  investigated  the  correlation  between  the  backgrounds  of  each  of  the 
crewmen  and  their  performance  ratings.  This  analysis  yielded  several  fascinating  points. 
First  and  foremost,  graduates  of  Undergraduate  Space  and  Missile  Training  consistently 
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had  better  CRM  related  attitudes  and  were  rated  higher  in  the  seven  Form  128  areas  than 
crewmen  who  did  not  attend  USMT. 


Table  2,  USMT  Attendance  and  CRM  Ratines 


GRPDYN 

ASSERT 

DECMAK 

STRESMGT 

MISHAP 

OVERALL 

OFFDUTY 

USMT  Pearson 

Correlation 

.285 

.308 

.245 

.151 

.168 

.240 

.045 

Sig.  (2-tailed) 

.002 

.001 

.008 

.110 

.079 

.010 

.637 

N 

113 

113 

114 

114 

110 

113 

112 

For  those  crewmen  who  did  not  attend  USMT  (this  includes  all  the  enlisted 
crewmen),  time  in  service  or  rank  greatly  affected  their  ratings  in  all  of  the  Form  128 
areas  except  the  overall  rating.  Correlations  between  education  and  ratings  in  the  Form 
128  areas  confirm  the  findings  with  the  USMT  and  the  Time  in  Service  /  Rank  variables. 
Phase  1  Discussion 

The  Mishap  Prevention  attitude  (SOCRMQ)  and  Stress  Management  performance 
(SORCRMQ)  had  a  correlation  of  .173  (significant  to  .086, 2-tailed).  Stress 
Management  is  a  key  element  in  Situational  Awareness  which  in  turn  is  a  key  element  in 
Mishap  Prevention.  Also,  Stress  Management  is  also  a  function  of  Workload  which  is 
clearly  a  function  of  whether  or  not  Automated  Systems  (part  of  Mishap  Prevention)  are 
understood  and  being  used  properly.  However,  as  can  be  seen  by  the  amount  of  narration 
necessary  to  describe  these  relationships,  this  is  not  an  easily  defined  construct  that  may 
not  prove  reliable  in  practice. 

Considering  graduates  of  USMT  are  all  officers,  and  officers  are  for  the  most  part 
better  educated  than  the  enlisted  force,  the  USMT  variable  may  not  mean  that  USMT  as  a 
program  does  a  better  job  of  training  CRM  concepts  than  the  all  enlisted  Technical 
School  does.  Indeed,  the  only  real  conclusion  is  that  officers  seem  to  receive  better  team 
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oriented  training  than  enlisted  members  do.  This  statement  ties  together  the  correlations 
seen  between  USMT  and  the  Form  128  areas  and  the  correlations  between  Education  and 
the  Form  128  areas. 

The  longer  enlisted  men  spend  in  the  military  and  the  higher  their  rank,  the  better 
they  are  rated  in  the  Form  128  areas.  This  seems  to  indicate  that  enlisted  crewmen 
eventually  learn  how  to  operate  in  a  team  as  they  spend  more  time  in  the  service. 
Interestingly  enough,  time  in  space  operations  and  time  in  position  do  not  affect  a 
crevmian’s  (officer  or  enlisted)  performance  in  the  Form  128  areas. 

Correlations  between  the  SOCRMQ  (attitude)  and  the  SORCRMQ  (performance) 
show  that  Assertiveness  with  Cronbach’s  alphas  of  .61  and  .66  on  each  questionnaire 
respectively,  and  a  correlation  of  .204  (significant  to  .04,  2-tailed)  is  the  only  reliably 
measured  attitude  related  to  performance. 
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IV.  Phase  2:  Guardian  Challenge  Crews  Methodology  and  Results 
Phase  2  Experimental  Goals 

The  goal  of  Phase  2  was  to  identify  those  attitudes  and  perceptions  held  by 
Guardian  Challenge  crews  that  correspond  to  their  overall  effectiveness/performance  as 
judged  by  Guardian  Challenge  evaluators.  Again  Phase  2  is  designed  to  capture  group- 
level  effects. 

Phase  2  Experimental  Design 

We  used  a  survey  to  measure  the  CRM  related  attitudes  of  Guardian  Challenge 
crews,  aggregated  individual  self-ratings  into  a  team  rating,  and  compared  these  attitude 
ratings  with  their  scores  from  the  competition.  Since  the  Guardian  Challenge  competition 
is  clearly  defined  as  fate  or  task  interdependent,  the  aggregated  rating  provided  an 
“effectiveness  measure. . .  taken  at  the  group  level”  as  recommended  by  Schmitt,  et  al. 
[1991]. 

Phase  2  Experimental  Instruments 

The  Space  Operations  Crew  Resource  Management  Questionnaire  was  translated 
for  use  by  Space  Lift  and  Missile  crews.  Primarily,  changes  were  made  to  the  1 1 
background  questions  to  accoimt  for  the  different  unit  and  crew  position  designations. 
Changes  to  the  actual  attitude  survey  items  were  cosmetic  (Flight  Commander  to  AFLC 
or  Satellite  to  Launch  Vehicle).  Some  questions  did  not  translate  well  between  the  three 
operations  fields  so  no  attempt  was  made  to  modify  them  and  crews  were  instructed  to 
ignore  any  question  that  they  did  not  believe  applied  to  them.  For  instance,  item  24  states 
that  “Even  when  fatigued,  I  perform  effectively  during  spacecraft  maneuvers.”  This 
question  clearly  does  not  translate  to  the  Missile  crews  since  if  their  spacecraft  were 
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maneuvering,  very  few  of  us  would  be  left  alive  to  interpret  the  results  of  this  study.  We 
opted  to  leave  the  questions  in  place  so  as  not  to  bias  the  results  between  the  surveys  by 
risking  the  insertion  of  new  concepts.  The  two  new  surveys  were  called  the  Space 
Operation  Crew  Resource  Management  Questiormaire  (Launch),  and  the  Missile 
Operations  Crew  Resource  Management  Questionnaire. 

Phase  2  Experiment 

The  SOCRMQ  was  mailed  or  faxed  to  each  of  the  Guardian  Challenge  crews.  Of 
the  nine  participating  Groups  only  one  Group  chose  not  to  participate  in  the  study.  Of  the 
approximately  50  participating  crewmen,  nine  crewmen  chose  not  to  respond  including 
the  entire  non-participating  Group.  Again,  this  yields  a  respectable  82%  response  rate. 
These  data  were  then  entered  into  SPSS  version  8.0  with  the  addition  of  each  crew  s  2041 
Score  (competition  score)  available  on  the  internet  at: 

www.vafb.af/mil/gc98/scores/98scores/98spce.html 
The  scores  are  presented  as  both  total  scores  or  a  percentage  of  the  total  possible  points. 
We  used  the  percentage  score  since  this  number  can  be  treated  as  a  normalized  score 
between  the  Space  Operations,  Space  Lift  and  Missile  Operations  crews. 

We  created  parameters  from  the  Form  128  areas  (Group  Dynamics, 
Communication,  Assertiveness,  Decision  Making,  Stress  Management,  Mishap 
Prevention,  and  Overall),  three  parameters  from  the  Factor  Analysis  of  the  SOCRMQ 
from  Phase  1  (Briefing,  Coordination  and  Monitoring)  and  also  calculated  the  Offduty 
parameter.  These  parameters  from  each  crewman  were  aggregated  into  team  ratings  for 
each  area  [Brannick,  et  al.,  1997:212]  and  then  correlated  against  each  team’s  competition 
(2041)  score. 
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Phase  2  Results 


The  results  were  imexpected  but  extremely  consistent.  The  higher  a  crew’s  score 
in  Guardian  Challenge,  the  worse  the  crew  rated  itself  in  CRM  attitudes.  To  ensure  there 
was  no  clerical  error,  the  raw  data  from  the  highest  scoring  crew  and  the  lowest  scoring 
crew  were  compared  by  hand  and  they  confirm  the  findings  in  Table  3. 

Tables.  Phase 2 Results 

COORD  MONITOR  BRIEF  GRPOYN  COMM  ASSERT  DECMAK  STRESS  MISHAP  OVERALL  OFFDUTY 

SCORE  Pearson  -.439  -.708  -.811  -.829  -.846  -.620  -.610  -.896  -.544  -.695  -.603 

Correlation 

Sig.  (2-  .102  .016  .007  .005  .002  .037  .040  .001  .103  .028  .043 

tailed) 

N  10  9889  9  9  97  8  9 

As  an  additional  check  of  the  results,  the  GC  data  was  further  divided  into  Space, 
Launch  and  Missile  operations.  Clearly,  statistical  significance  was  eliminated  due  to  the 
reduced  sample  size.  However,  Space  Crews  and  Missile  Crews  both  still  showed  the 
negative  correlations  while  Launch  Crews  showed  no  substantial  correlations. 

Phase  2  Discussion 

The  consistent  negative  correlation  should  not  be  interpreted  as  meaning  the 
worse  a  crew  is  at  CRM  the  better  it  does  in  competition.  Rather,  we  believe  the  results 
indicate  that  a  crew  that  can  accurately  identify  its  own  weaknesses  in  CRM  and  attempts 
to  compensate  for  these  weaknesses  will  perform  well  in  competition.  In  other  words,  for 
CRM  to  properly  work,  crews  must  understand  and  hold  themselves  to  a  higher  standard 
in  each  of  the  measured  CRM  areas.  Ultimately,  the  best  crews  never  believe  they 
perform  CRM  well  and  continuously  strive  to  do  better.  On  the  other  hand,  crews  that  do 
not  vmderstand  CRM  tend  to  rate  themselves  high  and  perform  low. 
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Another  possible  explanation  for  the  Phase  2  outcome  is  simply  that  the  Guardian 
Challenge  competition  is  an  unrealistic  environment  for  measuring  CRM  performance. 
Ultimately,  we  do  not  believe  this  to  be  true  since  the  Guardian  Challenge  (GC)  scenarios 
are  generated  by  each  participating  unit’s  Standardization  and  Evaluation  (DOV)  section 
and  are  submitted  for  a  common  GC  committee  approval.  Also,  GC  is  the  only 
environment  that  allows  for  a  substantial  amount  of  crews  to  be  tested  at  any  one  time. 
One  possible  future  improvement  may  be  to  use  the  results  of  the  Guardian  Challenge 
preliminary  competitions.  This  would  nearly  triple  the  number  of  participating  crews. 
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V.  Conclusion 


Implications  for  Researchers 

This  pilot  study  offers  a  first  look  at  the  application  of  CRM  to  the  Space/Missile 
Operations  arena.  Generally,  the  data  collected  seem  to  indicate  that  beyond  using 
Assertiveness,  attitude  measures  are  a  poor  indicator  of  whether  or  not  an  individual 
crewman  will  successfully  utilize  CRM.  Therefore,  attitude  measurement  is  a  poor 
performance  indicator  in  the  Space/Missile  Operations  environment.  Stated  another  way, 
the  CMAQ  does  not  translate  well  into  Space/Missile  Operations  and  should  not  be  used 
as  an  assessment  tool  by  instructors.  The  Form  128  may  still  serve  as  a  guide  for 
designing  a  CRM  evaluation  sheet  for  Space/Missile  crews,  but  the  sheet  does  not  seem 
well  suited  as,  or  easily  converted  into  a  self-assessment  sheet  to  be  used  by  individuals. 
Implications  for  Space  Operations  Training 

The  revelations  about  the  differences  between  officer  and  enlisted  member 
Assertiveness  are  interesting  in  themselves.  The  Air  Force  needs  to  do  a  better  job  of 
training  young  enlisted  men  to  serve  on  crews  with  senior  enlisted  and  officers.  In  each 
of  the  units  surveyed  for  Phase  1,  the  junior  enlisted  crewmen  are  the  ones  who  actually 
contact  and  command  the  satellites.  Officers  and  senior  enlisted  crewmen  serve  in  an 
‘advisory  role’  to  the  junior  enlisted.  It  would  be  very  unfortunate  and  extremely 
expensive  to  loose  a  satellite  because  a  junior  enlisted  crewmzin  (or  any  crewman  for  that 
matter)  was  afraid  to  contradict  a  senior  officer  who  misunderstands  a  situation  and 
requests  an  improper  command  be  sent  to  a  satellite. 
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Recommended  Further  Study 

Beyond  expanding  this  study  into  the  Guardian  Challenge  preliminary 
competition  mentioned  earlier,  another  interesting  investigation  would  be  to  validate  the 
AMC  Form  128,  or  its  recent  successor,  AF  Form  4031  (CRM  Skills  Training/ 
Evaluation  Form)  as  a  rating  sheet  for  Space/Missile  Operations.  This  could  be 
accomplished  by  exposing  every  crew  in  each  squadron  to  a  Guardian  Challenge  like 
scenario  as  part  of  monthly  recurring  or  quarterly  recurring  training.  At  the  same  time, 
independent  evaluators  could  then  rate  each  crew’s  CRM  using  one  of  the 
aforementioned  sheets.  The  rank  order  of  each  crew’s  performance  during  the  evaluation 
could  then  be  compared  against  their  CRM  ratings. 

Another  outcome  of  this  study  is  further  confirmation  of  Murray,  Weeks  and 
Siem’s  [1995]  and  later  Silverman,  Spiker,  Tourville  andNullmeyer’s  [1998]  Critical 
Incidents  approach  to  measuring  CRM  performance.  The  key  to  each  of  these  studies’ 
success  is  the  utilization  of  subject  matter  experts  to  create  a  construct  of  favorable 
mission/system  specific  CRM  behaviors.  Assessment  sheets  are  then  created  using  these 
core  behaviors.  These  assessment  sheets  are  used  by  evaluators  during 
simulator/operational  scenarios  to  create  a  realistic  model  of  good,  system  specific  CRM 
behaviors.  This  method  is  proving  to  be  far  more  reliable  than  attitude  measures  as  a 
predictor  of  good  performance. 
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APPENDIX  A: 


Space  Operations  Crew  Resource  Management  Questionnaire 


SPACE  OPERATIONS 
CREW  RESOURCE  MANAGEMENT 
QUESTIONNAIRE 


Department  of  the  Air  Force 
Air  University  (AETC) 

Air  Force  Institute  of  Technology 


Information 

_  About  this  Research  Study _ 

Thank  you  for  agreeing  to  participate  in  this  research  study.  Your  work  experience 
can  make  an  important  contribution  to  the  goals  of  this  study. 

Description  of  the  study:  The  goal  of  this  study  is  to  establish  a  baseline  for  Crew 
Resource  Management  (CRM)  attitudes  and  behaviors  in  Space  Operations.  To 
accomplish  this,  we  have  adapted  three  CRM  Attitude  and  Behavior  surveys  used  by  Air 
Mobility  Command  and  NASA  to  measure  the  performance  of  flight  crews. 

How  your  responses  will  he  used:  The  information  you  provide  will  be  correlated 
against  performance  measures  provided  to  us  by  your  raters  in  strict  confidentiality. 

Confidentiality  of  your  responses:  This  information  is  being  collected  for  research 
purposes  only.  NO  ONE  in  your  unit,  base,  or  MAJCOM  will  EVER  see  your  individual 
responses. 


PRIVACY  ACT  STATEMENT 

In  accordance  with  APR  12-35,  paragraph  8,  the  following  information  is  provided  as 
required  by  the  Privacy  Act  of  1 974. 

Authority:  10  U.S.C.  8012,  Secretary  of  the  Air  Force;  powers  and  duties;  delegation 
by;  implemented  by  APR  30-23,  Air  Force  Personnel  Survey  Program. 

Purpose:  To  improve  the  quality  of  training  given  to  Space  Operations  crews. 

Routine  Use:  Future  SO***  SPW  instructors  can  draw  upon  training  techniques  proven  to 
improve  effective  cockpit  crew  operations  to  better  train  space  operations  crews.  Data 
will  be  grouped  prior  to  analysis.  No  analysis  of  individual  responses  will  be  conducted 
and  ONLY  members  of  the  research  team  will  be  permitted  to  access  the  raw  data. 
Reports  summarizing  trends  in  crews  may  be  published.  No  crew  or  squadron  will  be 
identified  to  anyone  outside  of  the  research  team. 

Participation:  Participation  is  VOLUNTARY.  No  adverse  action  will  be  taken  against 
any  member  who  does  not  participate  in  this  survey  or  who  does  not  complete  any  part  of 
this  survey. 

Contact  Information 

If  you  have  any  questions,  please  feel  free  to  contact  me: 

Capt  John  Varljen  email:  jvarljen@afit.af  mil 

AFIT/  LAL  comm:  (937)  879  101 8 

2950  P  Street 

Wright-Patterson  AFB,  OH  45433-7765 
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Please  write  in  your  name  and  fill  in  the  appropriate  bubbles  on  the  computer 
answer  sheet.  Then  answer  the  following  questions,  filling  in  the  appropriate 
bubbles  on  the  answer  sheet. 

1 .  Current  Squadron 

1.  1  SOPS 

2.  2  SOPS 

3.  3  SOPS 

4.  4  SOPS 

5.  5  SOPS 

2.  Current  Crew 

1 .  Alpha 

2.  Bravo 

3.  Charlie 

4.  Delta 

5.  Echo 

3.  Current  Position  (note:  if  you  are  a  DCMDR,  mark  1) 

1.  FCMDR/ Crew  Chief 

2.  SSO 

3.  Payload  Specialist 

4.  Bus  Specialist 

5.  GSO/GC 

4.  Time  in  the  military 

1.  1-2  yrs 

2.  3-4  yrs 

3.  5-8  yrs 

4.  9-12  yrs 

5.  over  12  yrs 

5.  Time  in  Space  Operations 

1 .  0-6  months 

2.  7-12  months 

3.  13-18  months 

4.  19-24  months 

5.  25+  months 


6.  Time  in  Current  Position 

1.  0-6  months 

2.  7-12  months 

3.  13-18  months 

4.  19-24  months 
5. 25+  months 

7 .  Rank  (If  you  are  an  officer,  please  skip  to  the  next  question). 

1.  E1-E3 

2.  E4 

3.  E5 

4.  E6 

5.  E7-E9 

8.  Rank  (Please  skip  if  enlisted) 

1.  01 
2.  02 

3.  03 

4.  04 

5.  05 

9.  Did  you  attend  (please  respond  to  most  recent  of  the  following) 

1. USMT 

2. UMT 

3. UST 

4.  277x0  Tech  School 

5.  Other 

1 0.  Gender 

1. Male 

2.  Female 

11.  Education 

1 .  High  School 

2.  Some  College 

3.  Bachelor’s  Degree 

4.  Master’s  Degree 

5.  Doctorate 


Please  use  this  scale  to  answer  the  questions  in  this  section.  Record  your  responses 
on  the  computer  answer  sheet. _ 


Strongly 

Disagree 

Disagree 

Neither 

Agree 

strongly 

Agree 

1 

2 

3 

4 

5 

12.  Crew  members  should  avoid  disagreeing  with  others  because  conflicts  create  tension 
and  reduce  crew  effectiveness. 

13.  Crew  members  should  feel  obligated  to  mention  their  own  psychological  stress  or 
physical  problems  to  other  crew  persoimel  before  or  during  a  shift. 

14.  It  is  important  to  avoid  negative  comments  about  the  procedures  and  techniques  of 
other  crew  members. 

15.  Flight  Commanders  should  not  dictate  operations  procedures  to  their  crews. 

16.  Casual,  social  conversation  on  the  operations  floor  during  periods  of  low  workload 
can  improve  crew  coordination. 

17.  Each  crew  member  should  monitor  other  crew  members  for  signs  of  stress  or  fatigue 
and  should  discuss  the  situation  with  the  crew  member. 

18.  Good  communications  and  crew  coordination  are  as  important  as  technical 
proficiency  for  space  vehicle  safety. 

19.  Operators  should  be  aware  of  and  sensitive  to  the  personal  problems  of  other  crew 
members. 

20.  The  Flight  Commander  should  take  over  a  satellite  pass  in  emergency  or  non-standard 
situations. 

21.  The  SOO/SSO/PC  (pass  controller)  should  verbalize  plans  for  procedures  or 
maneuvers  and  should  be  sure  that  the  information  is  understood  and  acknowledged 
by  the  other  crew  members. 

22.  Crew  members  should  not  question  the  decisions  of  the  Flight  Commander  except 
when  they  threaten  space  vehicle  safety. 

23.  Crew  members  should  alert  others  to  their  actual  or  potential  work  overloads. 

24.  Even  when  fatigued,  I  perform  effectively  during  critical  spacecraft  maneuvers. 
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Strongly 

Disagree 

Disagree 

Neither 

Agree 

Strongly 

Agree 

1 

2 

3 

4 

5 

25.  Flight  Commanders  should  encourage  crew  members  to  question  procedures  during 
normal  operations  and  emergencies. 


26.  There  are  no  circumstances  when  the  Crew  Chief  (except  FCMDR’s  total 
incapacitation)  should  take  command  of  the  flight. 

27.  A  debriefing  and  critique  of  procedures  and  decisions  after  each  shift  is  an  important 
part  of  developing  and  maintaining  effective  crew  coordination. 

28.  My  performance  is  not  adversely  affected  by  working  with  an  inexperienced  or  less 
capable  crew  member. 

29.  Overall,  successful  operations  floor  management  is  primarily  a  function  of  the  Flight 
Commander’s  operations  proficiency. 

30.  Training  is  one  of  the  Flight  Commander’s  most  important  responsibilities. 

3 1 .  Because  individuals  function  less  effectively  under  high  stress,  good  crew 
coordination  is  more  important  in  emergency  and  abnormal  situations. 

32.  The  crew  changeover  briefing  is  important  for  vehicle  safety  and  for  effective  crew 
management. 

33.  Effective  crew  coordination  requires  crew  members  to  take  into  account  the 
personalities  of  other  crew  members. 

34.  The  Flight  Commanders  responsibilities  include  the  coordination  of  module  activities 
(i.e.  Repairs  to  Mod  XX). 

35.  A  truly  professional  crew  member  can  leave  personal  problems  behind  when  on  duty. 

36.  My  decision  making  ability  is  as  good  in  emergencies  as  in  routine  operations. 

37.  The  pre-pass  briefing  is  important  for  vehicle  safety  and  for  effective  crew 
management. 

38.1  use  an  operations  procedure  (checklist)  for  every  operation,  no  matter  how  routine. 
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Strongly 

Disagree 

Disagree 

Neither 

Agree 

Strongly 

Agree 

1 

2 

3 

4 

5 

The  following  questions  concern  your  crew  as  a  whole  team. 

39.  My  crew  establishes  guidelines  for  coordination  between  all  crew  positions.  The 
entire  crew  participates  in  briefings  as  a  team,  when  appropriate. 

40.  Crew  members  establish  and  maintain  a  team  concept  and  an  environment  open  for 
communications  (i.e.,  crew  members  listen  with  patience,  do  not  interrupt  or  “talk 
over,”  do  not  rush  through  the  briefings,  and  make  eye  contact  when  appropriate). 

41.  The  crew  ensures  the  operational  situation  matches  the  group  climate  (i.e.,  presence  or 
lack  of  social  conversation). 

42.  The  crew  ensures  non-operational  factors  do  not  interfere  with  necessary  tasks. 

43.  Crew  briefings  are  operationally  thorough,  interesting  and  address  crew  coordination 
while  planning  for  potential  problems.  The  crew  sets  expectations  on  how  to  handle 
deviations  form  normal  operations. 

44.  Crew  members  accept  performance  feedback  objectively  and  non-defensively. 

45.  Crew  members  openly  ask  questions  regarding  crew  actions  and  decisions. 

46.  Crew  members  “speak  up”  and  state  their  information  with  appropriate  persistence 
until  there  is  some  clear  resolution  and  decision. 

47.  When  conflicts  between  crew  members  arise,  the  crew’s  focus  remains  on  the 
problem  or  situation  at  hand. 

48.  Crew  members  listen  actively  to  ideas  and  opinions  and  admit  mistakes  when  wrong. 

49.  The  Flight  Commander  coordinates  operations  floor  activities  to  establish  a  proper 
balance  between  command  authority  and  crew  member  participation,  while  acting 
decisively  when  the  situation  requires. 

50.  Crew  members  clearly  state  operational  decisions  to  other  crew  members  and  receive 
acknowledgement. 

51.  The  “crew”  includes  all  crew  members  and  others  as  appropriate  to  conduct 
operations. 
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Strongly 

Disagree 

Disagree 

Neither 

Agree 

Strongly 

Agree 

1 

2 

3 

4 

5 

52.  The  crew  prepares  for  expected  and/or  contingency  situations  (i.e.,  adverse  weather 
at  remote  sites  or  preparing  for  orbital  maneuvers). 

53.  Crew  members  clearly  communicate  workload  and  task  distribution  and  receive 
acknowledgement  from  other  crew  members. 

54.  The  crew  allots  adequate  time  to  complete  tasks. 

55.  The  crew  prioritizes  secondary  operational  tasks  (i.e.,  dealing  with  tour  needs, 
command  post  commumcations,. . . )  to  retain  sufficient  resources  to  deal  effectively 
with  primary  flight  duties. 

56.  During  long  duty  periods,  crew  members  are  proactive  in  remaining  alert,  and  plan 
and  use  fatigue  countermeasures. 

57.  The  crew’s  actions  do  not  create  self-imposed  stress  and  additional  workload  (i.e.,  a 
late  satellite  contact  due  to  lack  of  situational  awareness/  planning). 

58.  Crew  members  recognize  and  report  when  their  duties  or  the  duties  they  observe 
others  performing  cause  an  overload. 

59.  The  crew  remains  calm  vmder  stress. 

60.  Crew  members  check-in  with  each  other  during  times  of  high  and  low  workload  to 
maintain  situational  awareness  and  to  remain  alert. 

61.  The  crew  establishes  guidelines  for  the  use  of  automated  systems  ( i.e.,  when  they  will 
disable  systems  and  when  they  must  verbalize  and  acknowledge  programming 
actions). 

62.  When  programming  demands  could  reduce  situational  awareness  and  create  work 
overloads,  the  crew  reduces  the  level  of  automation  or  disengages  automated  systems 
(i.e.,  manual  control  of  space  vehicle  or  disabling  automatic  ground  scheduling 
software). 

63.  The  crew  outlines  the  duties  and  responsibilities  of  crew  members  with  regard  to 
automated  systems  (i.e.,  attitude  control  computer  entries/uploads  and  cross  checking 
those  entries/  uploads). 
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Strongly 

Disagree 

Disagree 

Neither 

Agree 

Strongly 

Agree 

1 

2 

3 

4 

5 

64.  Crew  members  periodically  review  and  verify  the  status  of  space  vehicle  automated 
systems  (this  does  not  include  post  contact  analysis  by  staff). 

65.  Crew  members  verbalize  and  acknowledge  entries  and  changes  to  automated  system 
parameters. 

66.  The  crew  plans  for  sufficient  time  for  programming  of  spacecraft  attitude  control/ 
payload  computers  prior  to  maneuvers/  user  operations. 

67.  The  crew  consistently  assess  the  complexity  of  the  operating  environment  (remote  site 
weather,  communications,  maintenance  outages,  solar  flares,  eclipses,  etc.) 

68.  The  crew  assess  the  severity  of  abnormal  ground  or  space  systems  operation  and  other 
system  events  during  operations. 

69.  Overall,  the  crew  displays  technical  proficiency. 

70.  Overall,  the  crew  effectively  performs  the  mission. 

71.1  enjoy  spending  off-duty  time  with  my  crew. 

72.  I  often  spend  off-duty  time  with  my  crew. 

For  the  following  questions,  please  fill  in  the  appropriate  bubbles  on  the  computer 
answer  sheet. 

73.  Results  of  first  attempt  to  qualify  for  current  position? 

1 .  Unsatisfactory 

2.  Qualified 

3.  Highly  Qualified 

74.  Results  of  most  recent  recertification? 

1 .  Unsatisfactory 

2.  Qualified 

3.  Highly  Qualified 

4.  Too  new  for  recertification 

75.  Have  you  ever  received  a  real-world  critical  error? 

1.  Yes 

2.  No 
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APPENDIX  B: 

Space  Operations  Rater’s  Crew  Resource  Management  Questionnaire 
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SPACE  OPERATIONS 
RATER’S 

CREW  RESOURCE  MANAGEMENT 
QUESTIONNAIRE 


Department  of  the  Air  Force 
Air  University  (AETC) 

Air  Force  Institute  of  Technology 


Information 

_  About  this  Research  Study  _ 

Thank  you  for  agreeing  to  participate  in  this  research  study.  Your  work  experience 
can  make  an  important  contribution  to  the  goals  of  this  study. 

Description  of  the  study:  The  goal  of  this  study  is  to  establish  a  baseline  for  Crew 
Resource  Management  (CRM)  attitudes  and  behaviors  in  Space  Operations.  To 
accomplish  this,  we  have  adapted  three  CRM  Attitude  and  Behavior  surveys  used  by  Air 
Mobility  Command  and  NASA  to  measure  the  performance  of  flight  crews. 

How  your  responses  will  be  used:  The  information  you  provide  will  be  correlated 
against  crewmember  measures  provided  to  us  by  your  subordinates  in  strict 
confidentiality. 

Confidentiality  of  your  responses:  This  information  is  being  collected  for  research 
purposes  only.  NO  ONE  in  your  unit,  base,  or  MAJCOM  will  EVER  see  your  individual 
responses. 


PRIVACY  ACT  STATEMENT 

In  accordance  with  APR  12-35,  paragraph  8,  the  following  information  is  provided  as 
required  by  the  Privacy  Act  of  1 974. 

Authority:  10  U.S.C.  8012,  Secretary  of  the  Air  Force;  powers  and  duties;  delegation 
by;  implemented  by  APR  30-23,  Air  Force  Personnel  Survey  Program. 

Purpose:  To  improve  the  quality  of  training  given  to  Space  Operations  crews. 

Routine  Use:  Future  50*  SPW  instructors  can  draw  upon  training  techniques  proven  to 
improve  effective  cockpit  crew  operations  to  better  train  space  operations  crews.  Data 
will  be  grouped  prior  to  analysis.  No  analysis  of  individual  responses  will  be  conducted 
and  ONLY  members  of  the  research  team  will  be  permitted  to  access  the  raw  data. 
Reports  summarizing  trends  in  crews  may  be  published.  No  crew  or  squadron  will  be 
identified  to  anyone  outside  of  the  research  team. 

Participation:  Participation  is  VOLUNTARY.  No  adverse  action  will  be  taken  against 
any  member  who  does  not  participate  in  this  survey  or  who  does  not  complete  any  part  of 
this  survey. 

Contact  Information 

If  you  have  any  questions,  please  feel  free  to  contact  me: 

Capt  John  Varljen  email:  jvarljen@afit.af  mil 

AFIT/  LAL  comm:  (937)  879  1018 

2950  P  Street 

Wright-Patterson  AFB,  OH  45433-7765 
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Please  circle  the  appropriate  answer  for  each  question.  You  DO  NOT  need  to  fill  out  a 
computer  answer  sheet  for  this  survey. 


1 .  Current  Squadron 

1.  1  SOPS  2.  2  SOPS  3.  3  SOPS  4.  4  SOPS  5.  5  SOPS 

2.  Current  Position 

1.  DO  2.FCMDR 

3.  Current  Crew  (Do  not  answer  if  you  are  the  squadron  DO  or  DOO). 

1.  Alpha  2.  Bravo  3.  Charlie  4.  Delta  5.  Echo 

4.  Time  in  Space  Operations 

1.  0-6  months  2.  6-12  months  3.  12-18  months  4.  18-24  months  5.  24+ months 

5.  Time  in  Current  Position 

1.  0-6  months  2.  6-12  months  3.  12-18  months  4.  18-24  months  5.  24+ months 

6.  Rank 

1.01  2.02  3.03  4.04  5.05 

7.  Did  you  attend  (please  respond  to  most  recent  of  the  following) 

1.  USMT  2.  UMT  3.UST  4.  277x0  Tech  School  5.  Other 
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TEAMBUILDING 


Crewman  actively  participates  in  crew  briefings  when  appropriate. 

1.  Strongly  Disagree 

2.  Disagree 

3.  Neutral 

4.  Agree 

5.  Strongly  Agree 


For  each  person  you  directly  supervise,  write  the  number  in  Column  1 


RANK 

NAME 

ID 

IB 

IB 

ID 

IB 

IB 

IB 

B 

B 

BTil 

ID 

lEQ 

m 

lEQ 

m 

m 

lEQ 

m 

ira 

irn 

IQ] 

IB 

m 

■ 

■ 

■ 

IB 

IB 

IB 

IB 

B 

B 

IB 

IB 

IB 

IB 

IB 

IB 

IB 

IB 

IB 

■ 

IB 

IB 

IB 

IB 

Bl 

i _ 
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CREW  COMMUNICATION  ENVIRONMENT 


Crewman  listens  with  patience,  does  not  interrupt  or  “talk  over”,  does  not  rush 
through  briefings,  and  make  eye  contact  when  appropriate. 

1.  Strongly  Disagree 

2.  Disagree 

3.  Neutral 

4.  Agree 

5.  Strongly  Agree 


For  each  person  you  directly  supervise,  write  the  number  in  Column  2 


RANK 

NAME 

ID 

IB 

IB 

IB 

IB 

IB 

IB 

IB 

IB 

lEQ 

lED 

lEB 

m 

lEQ 

m 

IB 

lEQ 

IB 

IB 

m 

m 

m 

m 

IBi 
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OPERATIONAL  SITUATION 


Crewman  ensures  social  conversation  does  not  interfere  with  operational  tasks. 

1.  Strongly  Disagree 

2.  Disagree 

3.  Neutral 

4.  Agree 

5.  Strongly  Agree 


For  each  person  you  directly  supervise,  write  the  number  in  Column  3 


RANK 

NAME 

ID 

IB 

IB 

IB 

IB 

IB 

ID 

IB 

IB 

m 

IQ] 

lEB 

m 

m 

m 

m 

IIQ 

m 

IQ] 

m 

IB 

IB 

IB 

m 

■ 

■ 

■ 

■ 

IB 

IB 

IB 

IB 

IB 

IB 

IB 

IB 

IB 

IB 

IB 

IB 

IB 

IB 

IB 

IB 

IB 

IB 

IB 

_ 

_ 
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CREW  BRIEFINGS 


Crewman’s  briefings  are  operationally  thorough,  interesting,  and  address  crew 
coordination  while  planning  for  problems. 

1.  Strongly  Disagree 

2.  Disagree 

3.  Neutral 

4.  Agree 

5.  Strongly  Agree 


For  each  person  you  directly  supervise,  write  the  number  in  Column  4 


RANK 

NAME 

ID 

IB 

ID 

□ 

IB 

IB 

ID 

ID 

ID 

m 

lED 

IB 

m 

lEQ 

m 

m 

IB 

m 

m 

ID 

IB 

m 

m 

D 

B 

■ 
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■ 

IB 

IB 

IB 

IB 

IB 

IB 

IB 

IB 

IB 

IB 

IB 

IB 

IB 

IB 

IB 

IB 

IB 

IB 

_ 

B 

B 

Z 

1 

_ 

_ 
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PERFORMANCE  FEEDBACK 


Crewman  provides  positive  and  negative  feedback  to  fellow  crewmen  at  appropriate 
times. 

1.  Strongly  Disagree 

2.  Disagree 

3.  Neutral 

4.  Agree 

5.  Strongly  Agree 


For  each  person  you  directly  supervise,  write  the  number  in  Column  5 


RANK 

NAME 

D 

Q 

a 

D 

Q 

B 

D 

B 

B 

IQ 

cn 

m 

m 

m 

m 

m 

EQ 

EQ 

IQ 

n 

F73 

m 

■ 

■ 

■ 

■ 

■ 

■ 

■ 

■ 

■ 

■ 

■ 

■ 

■ 

■ 

m 

■ 

■ 

■ 

■ 

■ 

■ 

■ 

■ 

■ 

_ 

mmmmm 
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FEEDBACK  ACCEPTANCE 


Crewman  accepts  performance  feedback  objectively  and  non-defensively. 

1.  Strongly  Disagree 

2.  Disagree 

3.  Neutral 

4.  Agree 

5.  Strongly  Agree 


For  each  person  you  directly  supervise,  write  the  number  in  Column  6 


RANK 

NAME 

ID 

IB 

ID 

ID 

D 

ID 

ID 

ID 

D 

m 

im 

IB 

m 

m 

m 

m 

lEQ 

m 

m 

m 

m 

\m 

\m 

IQ] 

D 

B 

■ 

B 

■ 

ID 

B 

B 

IB 

IB 

IB 

IB 

IB 

IB 

IB 

IB 

IB 

IB 

IB 

IB 

IB 

IB 

IB 

Bi 

! _ 
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EFFECTIVE  INQUIRY 


Crewman  openly  asks  questions  regarding  crew  actions  and  decisions. 

1.  Strongly  Disagree 

2.  Disagree 

3.  Neutral 

4.  Agree 

5.  Strongly  Agree 


For  each  person  you  directly  supervise,  write  the  number  in  Column  7 


RANK 

NAME 

ID 

IB 

IB 

ID 

IB 

IB 

IB 

IB 

IB 

lEQ 

ID 

m 

m 

ID 

m 

m 

IQ 

m 

m 

m 

IB 

B 

m 

■ 

■ 

IB 

IB 

IB 

IB 

IB 

IB 

IB 

IB 

IB 

IB 

IB 

IB 

IB 

B 

IB 

IB 

IB 

IB 

IB 

IB 

IB 

IB 
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ASSERTIVENESS 


Crewman  speaks-up  and  states  his  information  with  appropriate  persistence  until 
there  is  some  clear  resolution  and  decision. 

1.  Strongly  Disagree 

2.  Disagree 

3.  Neutral 

4.  Agree 

5.  Strongly  Agree 


For  each  person  you  directly  supervise,  write  the  number  in  Column  8 


4^ 


RANK 

NAME 

ID 

IB 

B 

ID 

IB 

IB 

D 

IB 

B 

EQ 

ED 

lEQ 

ED 

lEQ 

ED 

ED 

lEQ 

lED 

ED 

in 

IB 

m 

IB 

■ 

■ 

B 

IB 

IB 

B 

B 

B 

B 

B 

IB 

IB 

IB 

IB 

IB 

B 

IB 

IB 

IB 

IB 

IB 

IB 

IB 

IB 

_ 
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CONFLICT  RESOLUTION 


When  conflicts  arise,  the  crewman’s  focus  remains  on  the  problem  or  situation  at 
hand,  listens  actively  to  ideas  and  opinions,  and  admits  mistakes  when  wrong. 

1.  Strongly  Disagree 

2.  Disagree 

3.  Neutral 

4.  Agree 

5.  Strongly  Agree 


For  each  person  you  directly  supervise,  write  the  number  in  Column  9 


RANK 

NAME 

D 

D 

Q 

a 

Q 

□ 

a 

D 

□ 

IQ 

m 

m 

m 

IQ 

m 

IQ 

m 

m 

IQ 

m 
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■ 

■ 

■ 

■ 

■ 

■ 

■ 

■ 

m 

■ 

■ 

■ 

■ 
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■ 

■ 

■ 

■ 

■ 

■ 
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_ 

1 
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i 

_ j 

j 

j 
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_ LEADERSHIP _ 

Please  answer  the  following  question  only  if  the  ratee  is  a  flight  eommander. 
Otherwise,  skip  to  the  next  question. 

The  flight  commander  coordinates  operations  to  establish  a  proper  balance  between 
command  authority  and  crew  member  participation;  while  acting  decisively  when 
the  situation  requires. 

1.  Strongly  Disagree 

2.  Disagree 

3.  Neutral 

4.  Agree 

5.  Strongly  Agree 


For  each  person  you  directly  supervise,  write  the  number  in  Column  10 


RANK 

NAME 

D 

Q 

B 
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B 
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B 

B 

gra 

{Q 

IB 

IQ 

m 

IB 

m 

IB 

IQ 

nil 

m 
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■ 
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B 

B 

B 

B 

B 
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B 

B 

B 

B 

B 

B 

B 

B 
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B 

B 

B 

i 

_ 
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_ CREW  PARTICIPATION _ 

The  Crewman  clearly  states  operational  decisions  to  other  crew  members  and 
receives  acknowledgement. 


1.  Strongly  Disagree 

2.  Disagree 

3.  Neutral 

4.  Agree 

5.  Strongly  Agree 


For  each  person  you  directly  supervise,  write  the  number  in  Column  11 


RANK 

NAME 

D 

Q 

D 

□ 

B 

B 

D 

B 

B 

Qij 

gn 

m 

EQ 

ED 

ED 

ED 

EQ 

ED 

gn 

m 
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CREW  PREPARATION 


The  crewman  properly  prepares  for  expected  and/or  contingency  situations. 

1.  Strongly  Disagree 

2.  Disagree 

3.  Neutral 

4.  Agree 

5.  Strongly  Agree 


For  each  person  you  directly  supervise,  write  the  number  in  Column  12 
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WORKLOAD  MANAGEMENT 


The  crewman  clearly  communicates  his  current  workload  level  to  other 
crewmembers.  He  allots  adequate  time  to  complete  his  tasks. 

1.  Strongly  Disagree 

2.  Disagree 

3.  Neutral 

4.  Agree 

5.  Strongly  Agree 


For  each  person  you  directly  supervise,  write  the  number  in  Column  13 
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TASK  PRIORITIZATION 


The  crewman  prioritizes  secondary  operational  tasks  to  retain  sufficient  resources 
to  deal  effectively  with  primary  operational  duties. 

1.  Strongly  Disagree 

2.  Disagree 

3.  Neutral 

4.  Agree 

5.  Strongly  Agree 


For  each  person  you  directly  supervise,  write  the  number  in  Column  14 
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FATIGUE 


During  long  duty  periods,  the  crewman  is  proactive  in  remaining  alert. 

1.  Strongly  Disagree 

2.  Disagree 

3.  Neutral 

4.  Agree 

5.  Strongly  Agree 


For  each  person  you  directly  supervise,  write  the  number  in  Column  15 
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SELF-IMPOSED  STRESS 


The  crewman’s  actions  (lack  of  action  or  errors)  do  not  create  additional, 
unnecessary  workload  for  himself  or  others. 

1.  Strongly  Disagree 

2.  Disagree 

3.  Neutral 

4.  Agree 

5.  Strongly  Agree 


For  each  person  you  directly  supervise,  write  the  number  in  Column  16 
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OVERLOAD  RECOGNITION 


The  crewman  recognizes  and  reports  when  his  duties,  or  the  duties  he  observes 
others  performing,  cause  an  overload. 

1.  Strongly  Disagree 

2.  Disagree 

3.  Neutral 

4.  Agree 

5.  Strongly  Agree 


For  each  person  you  directly  supervise,  write  the  number  in  Column  17 
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STRESS  LEVEL 


Crewman  remains  calm  under  stress. 

1.  Strongly  Disagree 

2.  Disagree 

3.  Neutral 

4.  Agree 

5.  Strongly  Agree 


For  each  person  you  directly  supervise,  write  the  number  in  Column  18 
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IB 


SITUATIONAL  AWARENESS 


The  crewman  checks-in  with  others  during  times  of  high  and  low  workload  to 
maintain  situational  awareness  of  operations  resources  (i.e.  antenna  activity,  who’s 
doing  what  with  which  satellite)  and  to  remain  alert. 

1.  Strongly  Disagree 

2.  Disagree 

3.  Neutral 

4.  Agree 

5.  Strongly  Agree 


For  each  person  you  directly  supervise,  write  the  number  in  Column  19 
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AUTOMATED  SYSTEMS 


The  crewman  establishes  guidelines  (or  utilizes  established  guidelines)  for  the 
operation  of  automated  systems  (i.e.,  when  he  should  disable  satellite  or  ground 
systems  and  when  they  must  verbalize  and  acknowledge  programming  actions). 

1.  Strongly  Disagree 

2.  Disagree 

3.  Neutral 

4.  Agree 

5.  Strongly  Agree 


For  each  person  you  directly  supervise,  write  the  number  in  Column  20 
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OVERALL  OBSERVATIONS 


Crewman  consistently  assesses  the  complexity  of  the  operating  environment 
(weather  at  the  remote  sites,  solar  flares,  PMI  activity,  etc.). 

1.  Strongly  Disagree 

2.  Disagree 

3.  Neutral 

4.  Agree 

5.  Strongly  Agree 


For  each  person  you  directly  supervise,  write  the  number  in  Column  21 
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OVERALL  OBSERVATIONS 


The  crewman  assesses  the  severity  of  abnormal  systems  operation  and  other  system 
events  during  operations. 

1.  Strongly  Disagree 

2.  Disagree 

3.  Neutral 

4.  Agree 

5.  Strongly  Agree 


For  each  person  you  directly  supervise,  write  the  number  in  Column  22 
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OVERALL  OBSERVATIONS 


Overall,  the  crewman  displays  technical  proficiency. 

1.  Strongly  Disagree 

2.  Disagree 

3.  Neutral 

4.  Agree 

5.  Strongly  Agree 


For  each  person  you  directly  supervise,  write  the  number  in  Column  23 
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OVERALL  OBSERVATIONS 


Overall,  the  crewman  effectively  performs  the  mission. 

1.  Strongly  Disagree 

2.  Disagree 

3.  Neutral 

4.  Agree 

5.  Strongly  Agree 


For  each  person  you  directly  supervise,  write  the  number  in  Column  24 
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